Support concatenated gzip (mgzip/pigz) layers in ztoc by prafgup · Pull Request #1871 · awslabs/soci-snapshotter

prafgup · 2026-02-23T01:46:36Z

Issue #, if available:
Fixes #1834. The C-level zinfo code only processed the first gzip member, causing all ztoc entries to map to span 0 for mgzip-compressed layers. This adds handling for multiple concatenated gzip members in all three functions: index generation, file-based extraction, and buffer-based extraction.

Description of changes:

Handle concatenated gzip members in generate_zinfo_from_fp by detecting additional data after Z_STREAM_END and resetting inflate state for the next member
- Ref - https://github.com/madler/zlib/blob/master/examples/zran.c#L221-L228
Handle concatenated gzip members in both extraction functions (extract_data_from_fp, extract_data_from_buffer) by manually skipping the 8-byte gzip trailer on the first member boundary (raw inflate mode), then switching to gzip mode (windowBits=47) which auto-consumes subsequent trailers
- Ref - https://github.com/madler/zlib/blob/master/examples/zran.c#L418-L436

Testing performed:

Added unit test -

lima sudo go test -v -count=1 -run TestDecompressWithPigz ./ztoc/...
=== RUN   TestDecompressWithPigz
=== RUN   TestDecompressWithPigz/span_size_64KB
=== RUN   TestDecompressWithPigz/span_size_256KB
=== RUN   TestDecompressWithPigz/span_size_1MB
--- PASS: TestDecompressWithPigz (0.53s)
    --- PASS: TestDecompressWithPigz/span_size_64KB (0.17s)
    --- PASS: TestDecompressWithPigz/span_size_256KB (0.17s)
    --- PASS: TestDecompressWithPigz/span_size_1MB (0.18s)
PASS
ok      github.com/awslabs/soci-snapshotter/ztoc        0.540s
PASS
ok      github.com/awslabs/soci-snapshotter/ztoc/compression    0.002s [no tests to run]
?       github.com/awslabs/soci-snapshotter/ztoc/compression/fbs/zinfo  [no test files]
?       github.com/awslabs/soci-snapshotter/ztoc/fbs/ztoc       [no test files]

lima sudo GO_TEST_FLAGS="-run TestSociZtocWithPigzLayers -count=1" make integration
cd cmd/ ; GO111MODULE=auto go build -o /Volumes/git/prafulg/soci-snapshotter/out/soci-snapshotter-grpc  -ldflags '-X github.com/awslabs/soci-snapshotter/version.Version=7d69a740.m -X github.com/awslabs/soci-snapshotter/version.Revision=7d69a7404b82eaeca6ccf229a1d4566aa920c8be.m  -s -w '  ./soci-snapshotter-grpc
cd cmd/ ; GO111MODULE=auto go build -o /Volumes/git/prafulg/soci-snapshotter/out/soci  -ldflags '-X github.com/awslabs/soci-snapshotter/version.Version=7d69a740.m -X github.com/awslabs/soci-snapshotter/version.Revision=7d69a7404b82eaeca6ccf229a1d4566aa920c8be.m  -s -w '  ./soci
integration
SOCI_SNAPSHOTTER_PROJECT_ROOT=/Volumes/git/prafulg/soci-snapshotter
=== RUN   TestSociZtocWithPigzLayers
=== PAUSE TestSociZtocWithPigzLayers
=== CONT  TestSociZtocWithPigzLayers
--- PASS: TestSociZtocWithPigzLayers (3.41s)
PASS
ok      github.com/awslabs/soci-snapshotter/integration 116.912

Tried converting/indexing the same image layer as [Bug] All index in ztoc have start_span and end_span 0 even with offset reaching 250mb+ #1834 that threw 0 start and end span,

{
  "version": "0.9",
  "build_tool": "AWS SOCI CLI v0.2",
  "size": 7262136,
  "span_size": 4194304,
  "num_spans": 118,
  "num_files": 15553,
  "num_multi_span_files": 80,
  "files": [
    {
      "filename": "app/var/data/conda-env/x86_64-conda-linux-gnu",
      "offset": 512,
      "size": 0,
      "type": "dir",
      "start_span": 0,
      "end_span": 0
    },
    {
      "filename": "app/var/data/conda-env/x86_64-conda-linux-gnu/bin",
      "offset": 1024,
      "size": 0,
      "type": "dir",
      "start_span": 0,
      "end_span": 0
    },
....
  {
    "filename": "app/var/data/conda-env/lib/libopenblasp-r0.3.30.so",
    "offset": 97394176,
    "size": 41424320,
    "type": "reg",
    "start_span": 23,
    "end_span": 32
  }
....
    {
      "filename": "app/var/data/conda-env/man/man1/bzegrep.1",
      "offset": 498103296,
      "size": 18,
      "type": "reg",
      "start_span": 117,
      "end_span": 117
    },
    {
      "filename": "app/var/data/conda-env/man/man1/bzcmp.1",
      "offset": 498104320,
      "size": 18,
      "type": "reg",
      "start_span": 117,
      "end_span": 117
    },
    {
      "filename": "app/var/data/conda-env/man/man1/bzmore.1",
      "offset": 498105344,
      "size": 4310,
      "type": "reg",
      "start_span": 117,
      "end_span": 117
    }
  ]
}

Manual test -

sudo soci convert --all-platforms xx/functions/user-function:0.2.0 xx/functions/user-function:0.2.0-soci

sudo nerdctl push --all-platforms xx/functions/user-function:0.2.0

-- clean all containerd and soci cache

lima sudo bash -c 'systemctl stop soci-snapshotter-grpc && cp /Volumes/git/prafulg/soci-snapshotter/out/soci-snapshotter-grpc /usr/local/bin/soci-snapshotter-grpc && cp /Volumes/git/prafulg/soci-snapshotter/out/soci /usr/local/bin/soci && systemctl start
  soci-snapshotter-grpc && sleep 2 && systemctl is-active soci-snapshotter-grpc


lima sudo nerdctl run --net host --rm --platform linux/amd64 --snapshotter soci --entrypoint sh xxxx/functions/user-function:0.2.0-soci -c "/app/var/data/conda-env/bin/python3.12 -c \"import sys; print(sys.version)\""

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Shubhranshu153 · 2026-02-23T23:38:09Z

@@ -228,6 +228,134 @@ func TestDecompressWithGzipHeaders(t *testing.T) {
 	}


needs a e2e test using pigz

I'll try to change TestDecompressConcatenatedGzip to use pigz (from os.exec) instead of using the current BuildConcatenatedGzipTar method for compression.

We can add integration test perhaps to test container with actual pigz tests, in some places we might be using some images from ecr public which we can populate with desired test images as needed.

I think I can get a minimal integration test working, which created a basic docker image locally with pigz compression and uses that to verify that ztoc get-file work fine as well as fuse mounts are present.

I think for now that should be enough and in future we can add some custom ecr image in soci-workshop-examples repository. This would probably needed to be done by some maintainer with access :)

I will update the PR soon.

@Shubhranshu153 I have update the unit test to use pigz for compression as well as added a minimal integration test which generates a pigz based image manually, indexes it and verifies the content of it.

Also I have manually ran and verified the working of the image which was resulting in 0 span ztoc creation in #1834

we can create a image with pigz in soci-workshop-example and have that test.
Its ok to have feature extension pending for future pr but ideally should have the required test for a new feature.

Shubhranshu153 · 2026-02-24T00:56:19Z

Overall looks correct to me. Reading more on gzip to see if i am missing something. Would like some e2e tests here though as we dont know if it works correctly through the both creation and execution lifecycle with a pigz compressor.

Signed-off-by: Praful Gupta <prafulgupta6@gmail.com>

prafgup · 2026-03-18T10:36:48Z

Hey @Shubhranshu153 just nudging regarding this :) is there something else I should be supporting with here? 😄

sondavidb · 2026-03-19T00:13:51Z

I'll take a look myself as well — sorry this slipped under my radar

prafgup marked this pull request as ready for review February 23, 2026 16:53

prafgup requested a review from a team as a code owner February 23, 2026 16:53

Shubhranshu153 reviewed Feb 23, 2026

View reviewed changes

Shubhranshu153 reviewed Feb 24, 2026

View reviewed changes

Comment thread ztoc/compression/gzip_zinfo.c

Shubhranshu153 reviewed Feb 24, 2026

View reviewed changes

Comment thread ztoc/compression/gzip_zinfo.c Outdated

prafgup force-pushed the prafulg/multiple-gzip-members branch from 7d69a74 to 680c963 Compare February 24, 2026 11:48

github-actions Bot added go Pull requests that update Go code testing Unit and/or integration tests labels Feb 24, 2026

Support concatenated gzip (mgzip/pigz) in ztoc generation and extraction

8ce5f10

Signed-off-by: Praful Gupta <prafulgupta6@gmail.com>

prafgup force-pushed the prafulg/multiple-gzip-members branch from 680c963 to 8ce5f10 Compare February 24, 2026 11:52

prafgup mentioned this pull request Feb 25, 2026

Port GzipZinfo to Go #1874

Draft

prafgup mentioned this pull request May 1, 2026

Upgrade cri-api for runtimeHandler parameter support #1943

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support concatenated gzip (mgzip/pigz) layers in ztoc#1871

Support concatenated gzip (mgzip/pigz) layers in ztoc#1871
prafgup wants to merge 1 commit into
awslabs:mainfrom
prafgup:prafulg/multiple-gzip-members

prafgup commented Feb 23, 2026 •

edited

Loading

Uh oh!

Shubhranshu153 Feb 23, 2026 •

edited

Loading

Uh oh!

prafgup Feb 24, 2026 •

edited

Loading

Uh oh!

Shubhranshu153 Feb 24, 2026

Uh oh!

prafgup Feb 24, 2026

Uh oh!

prafgup Feb 24, 2026 •

edited

Loading

Uh oh!

Shubhranshu153 Feb 24, 2026

Uh oh!

Uh oh!

Uh oh!

Shubhranshu153 commented Feb 24, 2026

Uh oh!

prafgup commented Mar 18, 2026

Uh oh!

sondavidb commented Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -228,6 +228,134 @@ func TestDecompressWithGzipHeaders(t *testing.T) {
		}

Conversation

prafgup commented Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Shubhranshu153 Feb 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

prafgup Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shubhranshu153 Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

prafgup Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

prafgup Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Shubhranshu153 Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Shubhranshu153 commented Feb 24, 2026

Uh oh!

prafgup commented Mar 18, 2026

Uh oh!

sondavidb commented Mar 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

prafgup commented Feb 23, 2026 •

edited

Loading

Shubhranshu153 Feb 23, 2026 •

edited

Loading

prafgup Feb 24, 2026 •

edited

Loading

prafgup Feb 24, 2026 •

edited

Loading